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1.     INTRODUCTION 

Hearn  and  Lawphongpanich  (1987)  studied  the  ascent  nature  of 
generalized  linear  programming  (GLP)  which  is  also  known  as  the 
dual  cutting  plane  or  primal  column  generation  technique.  In 
particular,  they  examined  the  properties  of  the  direction,  doLf, 
defined  as  the  difference  of  two  successive  dual  iterates 
generated  by  GLP  with  respect  to  the  Lagrangian  dual  function,  L. 
They  showed  that  iIglp  is  an  ascent  direction  for  L  at  points 
where  L  is  di f f erent iable .  At  nondi f f erent iable  points,  the 
column  entering  the  master  problem  is  not  unique  and  an  arbitrary 
choice  can  make  dctLp  a  nonascent  direction.  To  obtain  an 
additional  insight  on  the  ascent  nature  of  cIqlp  ,  they  also  showed 
that  doLF  is  a  solution  to  a  direction  finding  problem  for  L. 

In  this  paper,  we  introduce  a  new  class  of  direction  finding 
problems  and  showed  that  it  includes  the  one  which  produces  cIqlf. 
This  class  of  problems  contains  direction  finding  problems  which 
always  produce  ascent  directions  as  well  as  those  that  do  not, 
e.g. ,  the  doLF-  direction  finding  problem.  In  order  to  improve  the 
rate  of  convergence  of  GLP,  we  replace  the  cIqlf  direction  finding 
problem  with  another  from  this  new  class.  This  new  direction 
finding  problem  which  we  describe  below  guarantees  to  produce 
ascent  directions  and  is  similar  to  the  one  used  by  Demayanov  for 
the  minimax  problem.  Moreover,  a  line  search  step  is  also 
included  in  this  modification  of  GLP.  Hearn  and  Lawphongpanich 
(1987)  indicated  via  a  numerical  example  that  a  line  search  step 
reduces  the  number  of  iterations  for  GLP. 


For  the  remainder  of  the  paper,  we  state  GLP  algorithm  in 
both  the  cutting  plane  and  the  column  generation  form  and  define 
the  associated  Lagrangian  dual  problem  in  Section  2.  Then, 
Section  3  describes  the  new  class  of  direction  finding  problems. 
Finally,  we  present  the  modification  of  GLP  and  its  convergence 
analysis  in  Section  4. 


2.   PRELIMINARIES 

Consider  the  primal  problem: 

PI:        f*  =  min     f (x) 

x 

s  .  t .    g  (x)  *  b  , 

x  e  X , 

where  f  is  a  continuous  real-valued  function,  g  is  a  continuous 
function  from  Rn  to  Rn  ,  X  is  a  nonempty  compact  subset  of  Rn ,  and 
b  is  a  vector  in  Rm.   The  Lagrangian  dual  of  (PI)  is: 

Dl:        L*  =  maximize  {  L(u)  :  u  £  0 ,  u  e  RB  } 

where  L(u)  =  minimum  {f(x)  +  u[g(x)-b]  :  x  e  X} ,  and  xy  denotes 
the  usual  dot  product  between  vectors  x  and  y. 

When  L(u)  is  relatively  easy  to  evaluate,  the  pair  of 
problems,  (PI)  and  (Dl) ,  can  be  addressed  by  GLP .  Below  we  state 
the  algorithm  in  the  dual  cutting  plane  form  (Zangwill,  1969). 

The  Dual  Cutting  Plane  Algorithm 

Step  0 .   Find  a  point  x0  e  X  such  that  g(x0)  <  b.   Let  k  =  1,  and 
go  to  Step  1 . 

Step 1.   Solve  the  k-th  master  problem: 

Ml  :        max     w 
( w,  u) 

s.t.    w  <.    f(Xi)  +  u[g(xi)-b]     for  i  =  0 k-1 

u  £  0 


Let  (wk>Uk)  be  an  optimal  solution  and  go  to  Step  2. 

Step  2 .   Solve  the  k-th  subproblem: 

SI:  min{  f(x)  +  uk[g(x)-b]  :  x  e  X  } 

Let  Xk  be  an  optimal  point,  and  let  L(uk)  =  f  (xk)  + 
uk[  g  (xic) -b  ]  .  If  wit  =  L(Uk)  ,  uk  is  an  optimal  dual 
solution.   Otherwise,  if  wk  >  L(uk)  ,  then  replace  k  by 
k  +  1  ,  and  go  to  Step  1  . 

The  k-th  master  problem  (Ml)  is  a  linear  programming  problem 
with  the  following  dual: 


k-1 
M2 :        min    2   Tttf  (xt) 
n   i  =  0 

k-1 
s  .  t .   2   Jiig  (xi)  ^  b 
i  =  0 

k-1 

2  n,    =  1  , 
i=0 

nt  £  0      for   i  =  0 ,  .  .  .  ,k- 1 


When  (M2)  replaces  (Ml)  in  Step  1,  the  resulting  algorithm 
is  generally  known  as  Dantzi g-Wol f e  decomposition  (1960,  1961)  , 
column  generation  or  GLP .  Geoffrion  (1970)  also  classifies  the 
algorithm  in  the  cutting  plane  form  as  the  strategy  of  outer 
linearization  and  relaxation,  or,  in  the  column  generating  form 
as  the  strategy  of  inner  linearization  and  restriction. 


3.    A  CLASS  OF  DIRECTION  FINDING  PROBLEMS 

Consider  now  a  feasible  direction  scheme  for  solving  (Dl). 
At  a  given  feasible  point  Uk  and  a  parameter  a,  one  possible 
direction  finding  problem  is 

DF1:       Max     [  g  (xk)  -b  ]  ( v-uic) 
v 

s  .  t .    a    <.    L  (v)  , 

v  ^  0  , 

where  x*  is  a  solution  of  (SI)  ,  so  that  [g(xic)-b]  is  a 
subgradient  of  L  at  uk.  The  first  constraint  of  (DF1)  requires 
that  v  be  in  the  level  set  of  L  as  defined  by  a.  I f  a  >  LCu*) , 
any  feasible  solution  v  to  (DF1)  would  produce  an  ascend 
direction  of  the  form  v-uie,  and  the  best  value  for  a  is  L*  . 
Figure  3.1  illustrates  an  instance  of  (DF1)  in  which  a  is  set  to 
L(uic)   (=  7.0  )  and  the  optimal  solution  of  the  resulting  problem 
is  denoted  by  Vic .  Note  that  the  direction  vj<-uic  is  an  ascent 
direction  for  any  choice  of  subgradients . 

In  practice,  L(v)  must  be  approximated  since  it  is  not 
always  available  in  closed  form.  Each  different  approximation 
would  yield  a  different  direction  finding  problem.  Similarly,  the 
different  choices  for  a  and  x*  would  also  yield  different 
direction  finding  problems.  To  ensure  that  v^-u*  is  an  ascent 
direction,  the  approximating  function  along  with  the  value  for  a 
and  the  vector  xk  must  be  carefully  chosen.  Below,  we  examine  two 
different  direction  finding  problems  derived  from  (DF1). 


L(u)    =  7.0 
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Figure  3.1 :  An  illustration  of  direction  finding  problem  (DF1) 


First,  we  consider  the  direction  finding  problem  associated 
with  GLP  in  the  cutting  plane  form.  From  the  master  problem  (Ml)  , 
it  is  clear  that  the  approximation  of  L  is 

LA1(v)  =  min  {  f(x±)  +  v[g(x±)-b]:  i=0,...,k-l  }. 

Then,  Hearn  and  Lawphongpanich  (1987)  showed  that  u»c*i,  the 
solution  to  the  (k+l)-st  master  problem,  solves  the  following 
problem  derived  from  (DF1) : 

DF2:       max   [g  (x*) -b]  (v-uie) 
v 

S.t.     Wk+i  ^  Lai (v) , 

V  £  0  , 

where  wjc+i,  the  optimal  objective  function  value  of  the  (k+l)-st 
master  problem,  replaces  ot .  Define  X(uj«)  as  the  set  of  solutions 
to  (SI)  .  When  X(ujc)  is  a  singleton,  L  is  di  f  f  erent  iable  at  ux, 
and  [g(xjc)-b]  is  the  only  choice  of  '  subgradient '  for  the 
objective  function  of  (DF2) .  However,  when  X(uk)  is  not  a 
singleton,  L  is  nondi f f erent iable  and  the  choice  of  objective 
functions  becomes  infinite.  Moreover,  an  arbitrarily  chosen 
subgradient  could  result  in  (DF2)  generating  a  nonascent 
direction  as  shown  in  Hearn  and  Lawphongpanich  (1987) .  Below,  we 
consider  a  second  direction  finding  problem  in  which  [g(x*)-b]  is 
replaced  with  a  set  that  includes  the  subdi f f erent ial  of  L  at  Uk. 

Assume  that  X  can  be  represented  by  a  finite  discrete  set 
(yi, . . . ,yT) .  For  example ,  when  (PI)  is  a  bounded  integer  program, 
then  X  is  itself  a  finite  set  of  vectors  with  integral  elements. 


Then,  X(uic)  is  a  finite  discrete  set.  For  the  discussion  below, 
define  for  any  u  i  0 

X.(u)  =  (  yj :  f (yj) +u(g(yj) -b)  £  L(u)  +  e,  yj  e  X  }, 

that  is,  Xe(u)  is  the  set  of  all  e-approximate  solutions  of  (SI) . 
When  e  =  0,  Xo(u)  is  simply  the  set  of  solutions  to  (SI)  which  is 
previously  denoted  as  X(u) .  Since  L  is  concave,  the  e- 
subdi f f erent ial  of  L  is  well  defined  and  can  be  written  as 
follows  (see,  e.g.,  Clark,  1975;  Kiwiel,  1985;  Lemarechel,  1980; 
and  Zowe ,  1987) : 

6tL(u)  =  {  h:  L(v)  £  L(u)  +  h(v-u)  +  e,  V  v  £  0  } 

Similarly,  when  e  =  0,  <5oL(u)  =  6L(u)  ,  the  subdi  f  f  erent  ial  of  L 
at  u,  and  6L(u)  is  also  equivalent  to  conv{  [g(yj)-b]:  yj  e 
X(u)},  where  conv{-}  denotes  the  convex  hull  of  a  set.  Below, 
Theorem  3.1  relates  X«?(u)  to  the  e-subdi  f  f  erent  ial  of  L(u). 

Theorem  3.1:  Conv{  [g(yj)-b]:  yj  e  Xc(u)  }  is  a  subset  of  6cL(u) . 

Proof  :  For  every  v  £  0  and  for  each  yj  e  X«r(u) 

L(u)  +  [g(yj)-b] (v-u)  =  L(u)  -  Cg(yj)-b]u  +  Cg(yj)-b]v 

^  f  (yj)  -  e  +  [g(yj) -b]v 

£  L  (v)  -  e  ,  or 

L(u)  +  [g (yj ) -b] (v-u)  +  e  ^  L(v),  (3.1) 

where  the  first  inequality  follows  from  the  definition  of  Xc(u) 
and  the  second  inequality  from  the  definition  of  L.  Since 


equation  (3.1)  holds  for  all  yj  e  X«(u),  it  must  hold  for  all 
convex  combinations  of  yj  e  Xt(u),  and  the  theorem  follows. 

Given  the  above  definitions,  (DF1)  can  be  modified  as 
f ol lows : 


DF3 


where 


max  min  {  [  g  (yj  )  -b  ]  ( v-uk)  :  y„  e  X«(uic)  } 
s.t.       L(uic)  <,    LA2(v) 
v  £  0  , 

La2   =  min  {  f(x)  +  - v [ g (x) -b  ]  :  x  e  M*"1  U  {x0>  >,  and 

k-1 
M*-1  =   any  subset  of   U  Xc(ui) . 

i  =  l 


If  X(uic)  replaces  X«r(ujc)  in  the  objective  function  of  (DF3)  ,  the 
'min'  part  in  the  objective  is  the  expression  for  the  directional 
derivative  of  L  at  u*  in  the  direction  (v-uk) .  Note  that  (DF3) 
can  be  written  in  the  cutting  plane  format  as  follows: 


DF4: 


max  w 

s.t.         w  <.  [g  (yj  )  -b]  (v-uic) 

L(uic)  £  f  (x)  +  v[g(x)  -b] 

v  £  0  . 


V  yj  e  Xc(u*) 

V  x  e  M*"1  U  (xo) 


Under  the  assumption  that  X  is  a  finite  discrete  set,  L(u) 
can  be  equivalently  written  as  a  minimum  of  a  finite  number  of 
linear  functions.  Then,  (DF3)  and  (DF4)  resemble  the  direction 
finding  problem  of  an  algorithm  proposed  by  Demyanov  (see, 
Demyanov  and  Malozemov,  1974,  and  Lemarechel,  1980)  in  that  they 
require  a  full  knowledge  of  the  e -subd i f f eren t ial .  However, 


Demyanov's  direction  finding  problem  is  nonlinear  whereas  (DF4) 
is  1 inear . 


As  Stated  in  (DF3) ,  M*-1  can  be  any  subset  of  the  union  of 
the  previously  calculated  e-subdi f f erent ials .  Theoretically,  M*~ 1 
can  be  an  empty  set  and  the  algorithm  to  be  presented  in  the  next 
section  would  still  converge.  However,  taking  M*-1  as  an  empty 
set  means  that  L(v)  is  approximated  by  only  one  hyperplane 
defined  by  Xo.  In  practice,  this  may  not  yield  good  directions. 
Thus,  the  choice  of  the  set  M*-1  should  allow  a  good 
approximation  of  L(v)  and,  in  turn,  its  contours  in  the 
neighborhood  of  u* . 


4.    An  Ascent  Algorithm  and  its  Convergence 

The  algorithm  below  is  a  modification  of  the  cutting  plane 
algorithm  in  which  we  replace  the  master  problem  (Ml)  with  the 
direction  finding  problem  (DF3)  [or,  equivalent ly ,  (DF4) ]  and  add 
a  line  search  step. 

An  Ascent  Algorithm 

Step  0 .   Let  xo  e  X  satify  g(xo)  <  b.  Select  ui  £  0  and  compute 
XeCux).  Set  M°  =  0  and  k  =  1. 

Step  1  .  (Direction  Finding)  Solve  problem  (DF3)  (or, 

equivalent ly ,  (DF4))  and  let  w*  denote  the  value  of  the 
optimal  objective  function,  and  v*  denote  the  solution.  If 
wjc  ^  0,  stop  and  u*  is  an  optimal  solution.  Otherwise,  go  to 
Step  2. 

Step  2 .    (Line  search)  Solve 


L(u*  +  tk  ( Vic-Uk)  )  =  max  {  L(uk  +  t(vic-uie)):  0  £  t  £  1  } 

t 

and  set  uk+i  =  Uk  +  t*  ( vic-u*:)  .  Go  to  Step  3. 


Step  3 .    (Evaluate  L(Uk-i))  Solve  the  subproblem  (SI)  and 

construct  the  set  X«r(uic+i)  .  Set  k  =  k+1,  and  go  to  Step  1. 

In  Step  0,  the  point  xo  satifies  the  Slater  constraint 
qualification,  and  by  construction  xo  is  included  in  the 
approximating  function  LA2 .  This  prevents  Vk  from  being  unbounded 
since  v[g(xo)-b]  — >  -<»  as  if  any  component  of  v  goes  to  » .  For 
the  convergence  analysis  below,  it  is  assumed  that  the  set  (u:  ex 


£  L(u)>  is  bounded  for  all  a.  Thus,  uk  is  bounded  for  all  k  as 
wel  1  . 

The  first  three  theorems  justify  the  direction  finding 
problem  (DF3) . 

Theorem  4.1:  Uk  is  a  solution  to  (Dl)  if  and  only  if  u*  solves 
(DF3) . 

Proof :  For  convenience,  assume  that 

1)  X(u*)  =  {yi yp}  , 

2)  Xe(ujc)  =  (yi yP,yP*i y<*>  ,  and 

3)  M*"1  =  {xi Xc). 

where  p  £  q  and  q  <<    T. 

Let  ujc  be  a  solution  to  (Dl)  .   Then,  there  exists  a  (n,cx) 
satisfying  the  following  Karush-Kuhn-Tucker  (KKT)  conditions: 

p  m 

2  7tj[g(yj)-b]  +  2  arer  =  0 
j=l  r=l 

P 

2  TW  =1 

j  =  l 

c^Cu*]*-  =  0,    r  =  l,...,m 
n    and  ex  £  0 . 

where  e^  is  the  r-th  unit  vector  in  Rm  and  [zlx>    denotes  the  r-th 
component  of  the  vector  z.  Given  this  pair  of  multipliers,  (jt.cx)  , 
define  the  triplet  in'   ,&'  ,  ex '  )  as  follows: 

*j  s  rtj  i   J  =  1 P 


"j  =  0  .     J  =  P+l q 

SI  "    0,    i  =  1 c 

oC  =  ar  ,   r  =  1  ,  .  .  .  ,  m. 

Then,  (n'.&'.cc')  satisfies  the  following  KKT  conditions  for  (DF3) 
at  the  point  Un: 

q  c  m 

2  Jx^[g(yj)-b]  +  2  ei[g(xi)-b]  +  2  cCe,-  =  0 

j=l  i=l  r=l 

q 

2  ^  =1 

j  =  l 

pi [g(xi) -b]  =0 

aACuic]x-  =  0 

7i '  ,  &  '  ,and  a'  2    0  . 

Since  u*  is  feasible  to  (DF3) ,  Uk  must  be  optimal  to  (DF3)  as 
we  1 1  . 

Assume  that  ux  solves  (DF3) .  Then,  there  exists  a  in' ,p' , a ' ) 
satisfying  the  above  KKT  conditions  for  (DF3) .   Since  n '  and  p' 
are  nonnegative  and  ZjHj  =  1 ,  we  can  define 

a  =  SjAj  +  2ipi 

Txj     =    Jij/a,  j     =  1 p,p+l q 

&i    =    Pi/a ,  i    =  1  ,  .  .  .  ,s 

cx^    =    a^/a,  r    =  1 m. 

Then,  (n' ,5" , a " )  is  an  optimal  set  of  multipliers  at  the  point 
(z,v)  =  (L(Uk:)  ,ujc)  for  the  following  problem  which  is  related  to 
(DF4)  : 


DF5 :        max    z 

s.t.   z  £  f(yj)  +  Cg(yj)-b]v,    j  =  1 p,p+l,...q 

z  £  f(xi)  +  Cg(xi)-b]v,    i  =  1 a 

v  £  0  , 

that  is,  (L(uic)  ,ujc)  solves  DF5 .  However,  note  that  (Dl)  can  also 

be  written  as: 

D2 :        max    z 

s.t.   z  £  L ( v) 
v  £  0 . 
Then,  (DF5)  is  a  relaxation  of  (D2)  .  Since  (L(uk)  .u*)  solves 
(DF5)  and  is  feasible  (D2) ,  it  must  be  optimal  to  (D2) .| 

Theorem  4.2:  If  u*  does  not  solve  (Dl) ,  then 

min{  [g  (yj  ) -b]  (vic-uie)  :  yj  e  X«-(uk:)  }  >  0, 

where  v*  solves  (DF3) . 

Proof :  By  Theorem  4.1,  u*  does  not  solve  (Dl)  implies  that  u*  is 
not  a  solution  to  (DF3)  .  However,  Uk  is  still  feasible  to  (DF3)  , 
so  the  following  inequality  must  hold: 

min{  [g  (y  j  ) -b  ]  ( vjc-uic)  :  yj  e  Xc(ujc)  } 

=     max    {  min{  Eg (y j ) -b ] (v-u*) :  yj  e  Xc(ujc)  )) 
v  feasible 

>  min{  Eg  (yj  ) -b  ]  (uic-uic)  :  yj  e  Xc(uic)  >  =  0.| 

It  should  be  noted  that  the  above  two  results  also  hold  when 
La.2  (v)  is  replaced  by  the  actual  Lagrangian  function  L  in  (DF3)  . 


The  following  series  of  lemmas  and  a  theorem  demonstrate 
that  the  ascent  algorithm  above  converges  to  an  optimal  solution 
of  (Dl) .  Furthermore,  they  are  similar  to  the  standard  argument 
for  establishing  convergence  of  feasible  direction  algorithms  in 
nonlinear  programming. 

Lemma  4.3:  There  exists  an  e  >  0  such  that  if  u*  — >  u* ,  then 
XcCuic)  =  X(u*)  ,  for  k  sufficiently  large. 

Proof :  Let  X(u*)  =  (yx yP} .  Define 

i)  hi(u)  =  max  (f(yj)  +  u[g(yj)-b]:  j  =  1 p) 

ii)  h2(u)  =  min  (f(yj)  +  utg(yj)-b]:  j  =  p+1 T} 

iii)  ex  =  [hi (u*) +h2 (u*) ]/2 

At  u* ,  the  following  hold: 

A)  hi(u*)  =  L(u*)  =  f(yj)  +  u*[g(yj)-b],    j  =  l,...,p 

B)  hi (u*)  <  a  <  h2 (u*) 

C)  a  =  [a  -  hi (u*) ]  >  0 

From  the  continuity  of  hi(u) ,  h2(u) ,  and  L(u) ,  there  must  exist 
an  integer  K  sufficiently  large  so  that  for  every  k  £  K 

D)  |  hi  (uic)  -  hi  (u*)  |  <  a/2 

E)  |L(uk)  -L(u*) I  <  ct/4 

F)  for  j  =  1  ,  .  .  .  ,p 

| f (yj) +u^[g(yj) -b]  -  L(u») | 

=  |  f  (yj)+u,cCg(yj)-b]  -  f  (y  j  )  -u*  [  g  (y  j  )  -b  ]  | 
<  a/4  . 

G)  ha(uie)  >  a. 


From  (E)  and  (F)  ,  we  have  that  for  j  =  1  ,  .  .  .  ,p 

|  f  (yj  )  +uic[g  (yj  )  -b]  -L(Uic)  | 

<,     |  f  (yj) +uie[g(yj  ) -b]    -    L(u*)|     +     |L(u*)     -    L(u*)| 
£    a/2  , 

and    since    L(uj«)     ^    f(yj)     +u* [g (y j ) -b ] ,    V    j,     it    follows    that 

|  f  (yj) +uic[g(yj) -b]    -L(uic)  |     =    f (yj ) +u* [ g (yj ) -b  ]    -L(uj«)     <;    a/2,    or 
f  (yj  ) +u*[g(yj) -b]    ^    L(u*)+a/2. 

Thus,  for  any  e  between  (l/2)a  and  (3/4)a, 


f(yj)  +  uic[g(yj)-b]  £  L(uic)  +  e,   for  j  =  1, 


(4.1) 


that  is,  yj  is  an  element  of  X« (uk)  for  j  =  1 p.  Moreover, 

from  (E) , 

L(uie)   ^  L(u*)  +  a/4 

L(ujc)  +  e   ^  L(u*)  +  a/4  +  e 

<.  L  (u*)  +  a 

=  a 


<  f  (yj)+uic[g(yj)-b]  ,    for  j  =  p+l,...,T.     (4.2) 


where  the  third  inequality  follows  the  above  selection  of  e,  the 
equality  from  (C) ,  and  the  last  inequality  from  (G)  and  the 
definition  of  ha(u) .  This  means  that  yj  does  not  belong  to 
X«r(u*),  for  j  =  p+l,...,T.  Therefore,  Xe(uk)  =  X(u»). 


Lemma  4.4:  For  a  given  direction  d, 

min{  g"d:  g"  e  conv ( [ g (yj ) -b ] :  j  =  1 p)  } 

=  min{  [g(yj) -b]d:  j  =  1 p  } 

Proof  :  The  result  follows  from  the  fact  that  the  problem  on  the 
left  hand  side  can  be  stated  as  a  linear  program 


min    2  (Cg(yj)-b]d)nj 
j  =  l 

P 

S  .  t  .  2  Ttj   =   1 

j  =  l 

Kj   ^   0,    j   =   1  , 


which  always  yields  an  extreme  point  solution. 

Lemma  4.5:  If  Xc(uj<:)  =  (yi yp)  and  min{  [g(yj)-bjd:  j 

l,...,p  }  >  0,  then  there  exists  a  t  >  0  such  that 

L(ujc  +  ad)     £    L(uk)+amin{     [g(yj)-b]d:     j     =     l,...,p    } 
for    all    0    £    a    <.    t. 

Proof :   Assume  without  loss  of  generality  that 

0  <  [g(yi)-b]d  =  min{  [ g (y j ) -b ]d :  j  =  1 p  } 

Def  ine 


t  =  mm 


{f  (yj  )  +u[g  (y  j  )  -b  ]  -  L(uic)>     for  all  j  such  that 

,g(yi)d  >  g(yj)d  &  j  t    Xc(uk) 

Cg(yi)  -  g(yj )  3d 


and  observe  that  t  >  0  since 


f(yj)  +  u[g(yj)-b]  >  L(uic)       for  j  t    Xr(uk), 


and  [g  (y  i ) -g  (yj  )  ]d  >  0  by  construction.  Thus,  for  any  0    <.    <j    <.    -r 
and  every  j  %    X«.(uie)  and  g(yi)d  >  g(yd)d 

(f(yj)  +  ujc[g(yj)-b]  -  L(u»c) 
a  £   


[g  (yi)  -  g(yj)  ]d 
a[g(yi)-b  -  g(yj)+b]d  <.       f  (y j  )  +  u*  [g  (y  j  ) -b  ]  -  L(u*) 

L(uic)  +  a[g(yi)-b]d  <>    f(yj)  +  (ujc  +  ad)  [  g  (uj  )  -b  ]        (4.3) 

However,  for  j  t    Xc(u>c)  and  0  <  [g(yi)-b]d  ^  [g(yj)-b]d 

L(ujc)  <  f(yj)  +  uicCg(yj)-b] 
LCujc)  +  or[g(yi)-b]d  <  f(yd)  +  (uk  +  ad)  [g  (yd  )  -b  ]        (4.4) 

and  for  j  e  Xc(uic) 

L(uk)  <:  f(yj)  +  uic[g(yj)-b] 
L(u!c)  +  <x[g(yi)-b]d  ^  f(yj)  +  (u,c  +  crd)  [  g  (yj  )  -b  ]  .      (4.5) 

Combining  (4.3)  ,  (4.4)  and  (4.5)  ,  we  have  that  for  0  £  a  £  t 

L(ujc)  +  a[g  (yi) -bid  <>    min{  f (yj )  +  (u*  +  ad)  [ g (yj ) -b  ]  :  j  =  l T) 

=  L  (ujc  +  ad)  . 

Lemma  4.6:  Assume  that  e  is  chosen  as  in  Lemma  4.3  and  the 

algorithm  generates  a  sequence  {u*} .  Then,  there  cannot  be  a 
subsequence  (ujc)  ,  k  e  Q,    with  the  following  properties: 

i )   Uk  — >  u~ ,      k  e  Q  , 
i  i )   Vk  — >  Vo» ,      k  e  Q  ,  and 
iii)   min{  g  "  (v«,-u»)  :  g"  e  conv  (  [  g  (y ) -b  ]  :  y  e  X(u<=))  }  >  0, 

where  Q  is  a  subset  of  {1,2,3,...}. 


Proof :  Assume  that  X(u~)  =  (1 p} .  By  (iii)  and  Lemma  4.4, 

min{  g"(v.-u.):  g"  e  conv  (  [g  (y) -b  ]  :  y  e  X(u=»))  } 

=  min  {  [g  (yj  )  -b]  (v~-u«)  :  j  =  l,...,p  }  =  &  >  0. 

From  Lemma  4.3,  there  must  exist  a  K1  sufficiently  large  such 
that  Xc(uic)  =  {l,...,p}  for  some  k  £  K1  and  k  e  ft.  Since  the  set 
{l,...p>  is  finite,  there  must  exist  a  index  j*  such  that 

j*  =  arg  min{  [ g (y j ) -b ]  ( vk-uk)  :  j  =  1 p  }  (4.6) 

infinitely  often.  Define  ft1  to  be  the  subset  of  ft  for  which  j*  is 
the  index  which  yields  the  minimum  value  for  the  right  hand  side 
of  (4.6) .  For  convenience,  we  assume  that  j*  =  1.  Then,  we  have 
that 


Lim   Cg(yi) -b]  (vic-uic)   =   [g(yi)-b]  (v.-u.)  =  B  >  0, 
k  e  ft1 

and  it  follows  that  there  exists  K2  £  K1  such  that 


Cg(yi)  -b]  (vic-uic)  >  &/2    for  k  *  K2  and  k  e  ft1.  (4.7) 

Thus,  at  uic  the  direction  (vjc-ujc)  is  an  ascent  direction. 
Moreover,  since  Uk+i  maximizes  L(u)  along  the  direction  (Vk-Uk) , 

L(uic+i)  ^  L(ujc  +  cr(vic-uje))   for  0  <  a  <  t  and  k  e  ft1 

£  L(ujc)  +  a[g(yi)-b](Vk-uk)     for  0  <  a  <  t  and  k  e  ft1 
2    L(uic)  +  (ffB)/2     for  0  <  a  <  t  and  k  e  ft1,      (4.8) 

where  t  is  as  defined  in  Lemma  4.5  and  the  last  two  inequalities 
follow  from  Lemma  4.5  and  equ.  (4.7)  .  Letting  k  approach  infinity 
on  the  subsequence  ft1,  (4.8)  yields 


L(u.)  £  L(u-)  +  (ct&)/2 

which  is  a  contradiction  since  both  a  and  &  are  positive.! 

Theorem  4.7:  Assume  that  e  is  chosen  as  in  Lemma  4.3.  If  the 

algorithm  generates  a  sequence  {u*} ,  then  it  must  converge 
to  an  optimal  solution  of  (Dl) . 

Proof :  Assume  that  the  algorithm  generates  an  infinite  sequence 
of  points  and  none  of  which  is  a  solution  to  (Dl)  .  Since  (uic) 
lies  in  a  bounded  region,  there  must  exist  a  convergent 
subsequence,  i.e.,  uic  — >  u«» ,  for  k  e  Q,  where  Q    is  a  subset  of 
{1,2,3,...}  and  u~  does  not  solve  (Dl). 

Assume  that  X(u«)  =  {l,...,p}.  By  Lemma  4.3,  Xe(ujc)  =  X(u») 
for  k  sufficiently  large,  i.e.,  k  £  Ki,  thus  Xc(u<=)  =  X(u<=). 
Since  Uo>  is  not  a  solution  to  (Dl)  , 


0  <  &•  =  max  min{  [g(yj)-b](z- u» ):  j  =  l,...,p  } 
z    j 
s.t.        L(u-)  £  L  (z) 

z   £  0 . 


For  k  £  Ki ,  def  ine 


DF6  :       &jc  =  max  min{  [g  (yd )  -b]  (z-Uk)  :  j  =  l,...,p  } 

z    j 
s.t.       L(ujc)  £L(z) 

z   £  0 . 


Note  that  &*  varies  continuously  with  Uic .  Thus,  &*  — >  6=  for  k  e 
Q  and  k  £  Ki . 


Now,  let  vie  be  a  solution  to  (DF3)  defined  at  u* .  Then,  by 
construction,  v*  is  bounded  and  there  must  exist  a  subset,  Q1,  of 
Q    such  that  Vk  — >  v<=  on  Q1.   Moreover,  since  (DF3)  is  a 
relaxation  of  (DF6) , 

&*  £  min{  [g(yj) -b]  (vic-uic)  :  j  =  l,...,p  },  (4.9) 

for  k  £  Ki  and  k  e  Q1.  From  the  finiteness  of  of  the  set 
{l,...,p},  there  must  exist  an  integer  j*  such  that 

j*  =  arg  min{  [ g (yj ) -b ] ( v*-uk) :  j  =  l,...,p  }  (4.10) 

infinitely  often.  Let  Q2  be  a  subset  of  Q1  for  which  j*  is  the 
index  which  minimizes  the  right  hand  side  of  (4.10)  and  for 
convenience  assume  that  j*  =  1.  Then,  combining  (4.9)  and  (4.10) 
gives 

[g  (yi)  -b]  (vic-ujc)  ^  6*      for  k  £  Ki  and  Q2  . 
Taking  the  limit  on  both  side  with  respect  to  Q2  ,  we  have  that 

lim  [g  (y  i) -b]  (vjc-ujc)  =  [  g  (y  i )  -b  ]  ( v--u«)  ^  3~  >  0 

which  contradicts  Lemma  4.6. 

Therefore,  if  the  algorithm  terminates  after  a  finite 
number  of  iterations,  Theorem  4.1  and  4.2  guarantee  that  Uk 
solves  (Dl) .  Otherwise,  the  algorithm  generates  an  infinite 
sequence  which,  by  Theorem  4.7,  converges  to  an  optimal  solution 
of  (Dl) .  Also,  it  is  interesting  to  note  that  although  the 
algorithm  uses  e -subdi f f eren t ial  in  calculating  its  ascent 
directions  an  exact  optimal  solution  can  be  obtained  by  choosing 


e  correctly.  In  general,  one  expects  algorithms  using  an  e- 
subdi f f erent ial  to  produce  E-optimal  solutions,  po3Sibly  in  a 
finite  number  of  iterations.  We  consider  this  type  of  algorithms 
in  a  separate  study  to  appear  later. 
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